And here we are, with another day of work ahead. And another huge unchecked/dinstall run about to finish. Yesterday evening Ansgar and later on Phil turned up, so we are now 5 persons sitting here and mostly coding happily away. Mark continued trying to wrap his brain around the changes we need to properly support multiple archives, with different sets of rights attached to them. It is a complex topic giving lots of headache. Besides that he is on a very good way of letting us completly remove apt-ftparchive by importing all the data we need to write Packages/Sources files on our own. And even better, the same work we can use to rewrite our rm and cruft-report commands in a sane way, not parsing Packages files anymore. Torsten did help with various database issues around the Packages/Sources issue and continued working on importing all the source filenames into the database, so we can soon provide a Contents file for sources. My own work, besides writing blog entries and preparing meeting minutes, consists of a rewrite of the script generating our Release files, the old one just wasn t up to it anymore. Especially not as we are on our way of having more and more of our tools use parallel processing, to fully use the power our ftp-master host has. And when I am done with this I will be off to tackle the keyring maintenance part we will need to enable autosigning by buildds to work. Should get interesting, with the shit of an interface one usually finds to gpg Ansgar started out by working on changes to our NEW handling, something I started with the dpkg bug #619131 and he just finished our site of the NEW handling scripts. Which means that, as soon as the dpkg maintainers added the code on their side and support this field in the .dsc files, NEW handling for packages like linux-2.6 will be much smoother. He is now implementing version control constraints into our control-suite tool. Phil is working on wanna-build stuff. And while this is just a short overview of what we did, there is always lots of discussion and bugfixing going on. It is incredibly useful to have a team sitting in one room for some days.

Good morning, and welcome to another day of archive damage, i mean hackery. Well, at least at the time of this writing it is morning and we are starting off into another day of work. As we promised to try to have at least a once-a-day-installation run I am currently processing the queue of uploaded packages, around 4gigabytes from 431 changes files. After that a dinstall will follow to push that all out. Lets see how often we can do this again today, yesterday we managed multiple times. Yesterday my work consisted of getting the ries sync into something nice and acceptable, starting to think of the change to only use the suite codenames in future (long term making lots of code issues easier, but short term a good headache) and doing the groundwork so we can upgrade the security and backports archive to squeeze/pg9 too. Mark finished of the Built-Using flag and then moved on to think about database file tracking changes related to multiple archives/locations. Which sounds simple in so few words but is definitely not simple at all. Torsten started to work on source contents and also on generating the packages/sources files out of our database. Both together is taking a good amount of time, so he continues to work on those today. In the evening we had a meeting with the Debian Hurd porters about the future of their port within the Debian archive. The perceived Hurd situation until then was that every port has to follow certain rules or gets thrown out but Hurd doesn t seem to ever get into a releasable state . Well, we did get a clean view on both sides standpoint and in the end emerged with a solution which works for both sides: The Hurd port stays on FTPMaster until Wheezy is released. Should they have managed to get the port into a state that it is released together with all the others (probably as a technology preview), it is kept in the archive. Should they not manage this, the port will be removed from the main archive and move fully to debian-ports.org. It may then reenter the main archive whenever it is ready to get released with the next release. For today we are all continuing in our parts, Mark is yelling at the insanity of the multi-location thing (something we DO need when we want to merge multiple archives, which we do want), Torsten curses the code for Packages/Sources and while I hope that DSA can find time today to upgrade our security- and backports-archive machines to Squeeze, I try to wrap my head around the code we kindly got provided by Serafeim Zanikolas - the dinstall programming fun task I asked a while ago. Im not sure yet I will succeed. :) And then we will get more arrivals this evening, so more people doing stuff tomorrow.

I am at the next edition of Debian FTPMaster meetings in the LinuxHotel Essen currently. We arrived Sunday evening and will stay until 27th March, giving us again a nice week of time to work on outstanding archive issues. And our todo does list a good number of points. Currently we are 3 persons, all 3 FTPMasters. On Wednesday evening one wanna-build member will join and on Friday evening one FTPTeam member arrives for the weekend. While thats not all of our team it still gives us a good base to work on. And so we already did, starting Sunday evening with unbreaking some fallout from Lenny->Squeeze update of the ftpmaster host as well as discussing some of our agenda points and how to tackle them. Today I started changing our backup strategy. Instead of dumping the whole dak database at dinstall start and end, a 4gb file each time, later compressing and then expiring them, we don t dump them at all anymore. Instead we save the postgres transaction id and let the Debian Sysadmin team do the backup. Which they do anyways, using the PostgreSQL WAL stuff. Which means we can just tell them to please reset the dak db to the place of transaction $BLA . Something that only works as we are switching to Postgres 9.0 and will use replication to sync our DD-accessible machine. Torsten started work by explaining us his ORMObject idea in our database layer, then violently vomitting when I pointed him at some code to fixup. Seems that code isn t all the most beautiful one, so he switched to cleanup in a different place. Lets see if I get him back to that bad code. Mark is playing with our database and trying to get that from 8.4 to 9.0, seems like some stuff we are using isn t all that en vogue anymore in 9.0, so might need adjustments.

And so there went a day of work. Minutes done, come back tomorrow

---

Okok, more text: We did lots of stuff. There was breakfast, soon after we had Lunch, and guess, Dinner also. Now, would you please come back tomorrow?

---

Anyways, we did some things. One of those was to have PostgreSQL 9.0 and to enable the replication to the machine any DD can access. Which is, from today on, ries.debian.org. And for the first time ever (since that time ftp-master got closed to the public), DDs can now see the live status of the archive, literally a second after a package got accepted a dak ls on ries will show you that fact. Torsten worked on fixing up old remnants of code and database entries around contents and also tackled a problem between our usage of SQLalchemy and the python multiprocessing module. He also started working on the source contents and was weird enough to get us standardized logging into a dak tool. Mark worked with DSA on the Postgres 9.0 upgrade to fix what was to fix in out database and did some tests with our environment in squeeze and coordinated changes in python-apt needed for multiarch and xz. He then started working on some larger database changes around the built-using feature. I myself started off with getting generate-packages-sources to behave when there is an error instead of silently ignoring that. I also did a bit of groundwork to enable us to do NEW processing adding overrides for all affected binary packages, not just those from the architecture currently uploaded. That will allow stuff like linux-2.6 to pass NEW just once, not once per architecture. That involves a new field in our dsc files and the status of that is tracked in Bug #619131.

So, this should be the first blog post of the DebConf6 Live Blog , run by the DebConf6 Organizers, which appears on Planet Debian. Lets start with some pictures: I have put up some pictures from my flight (the food :) ), and then from today here in Oaxtepec online. You can find them at our gallery, below the DebConf6 album. Everyone else who takes picture: Feel free to register at the gallery, and then send a little mail to gallery-admins@debconf.org, then you can put all your pictures also online there. On the network side - well, we dont have network up in the Vacational Centre yet, Im sitting in a Internet Cafe down in the town, where some locals are building up Wifi Stuff to get Net to our events place. Lets see how that works out. For the rest: Its a very nice place, really.

I ve been using dailystrips since April 2006 to get a daily overview page of all the various daily web comics I (and a few friends) like to see. It makes it much simpler than having to browse dozens of various webpages, with all their useless webbugs integrated, to see stuff. That had some drawbacks:

• dailystrips is unmaintained code. Last upstream release is from 2003.
• It is uncapable of fetching any other information than just the image. Which is bad for those like xkcd, Astronomy picture of the day or others where the title/alt text actually is an important part of it.

So about two or three weeks ago I took a look at dailystrips code to figure out where I need to stuff some lines in to get the additional information. Yuck. No. I don t like it. Not because it is perl, that alone would be fine. So, instead I went and did a little rewrite of it. In a simple and pure bash script. I duplicated the functionality of the daily mode (fetch images per day and put them into a daily subdirectory, create html files for them) and didn t bother with the fetch a random strip one as I never used that. Turned out at 276 lines of shell script that can even fetch alt/title and not break on funny characters in them, yay. Even better, as the search terms for the additional information is just a regex using bashs capability there run over the html file, one can have anything from it put as alt/title onto the final page. When I had that working I asked a colleague to provide a bit of CSS, so it also looks better now. An example output is available (note that the Previous/Archive links are not available there), and if you want you can take a look at the code too. There is no Debian package and I m not sure there should ever be one. If you want to get me patches feel free to mail me, be it for the CSS or the script, whatever.

So after quite a while I again had the opportunity to do the ftpmaster works of a Debian release. Not quite my first and hopefully not my last one, lets see. And, like last time, Mark joined to help with the work. Again, a few moments and not a detailed log (times in UTC):

• [08:30] I ll start the ftpmaster side by turning off all cronjobs, prepare some database backup, get myself sorted.
• [09:33] Got dak.git prepared with most of the changes we need for the release
• [10:02] Oldstable suite created, load the contents from stable into oldstable. Lenny is now oldstable
• [11:00] Load stable with the contents of testing. That is, stable now is squeeze.
• [11:35] Cleanup various dirs in dists/, like - old installers, some breakage from elsewhere, fixup READMEs.
• [11:52] Have Lenny Release file regenerated and signed
• [12:29] Have Squeeze Release file generated and signed
• [12:54] Push the cd builder machine so they can start building our CD/DVD images
• [13:00] Start a manual dinstall run to check through all the rest of the archive if it still works
• [13:43] Get a heart attack when clean-suites reports Cleaned 2543 files, 2500 MB , but then discover it is only the hppa removal that got in just late before the release. Puuh.
• [14:30] all of the manual dinstall run is done, everything fixed, the main archive looks nice. So lets start breaking security
• [15:40] Yay, security archive all done, waiting for the general release to push the world
• [17:30] After a short break I updated the backports.debian.org archive also, it is now ready to receive uploads targetting squeeze (but wait for a mail from the bpo team!), the lenny-backports suite is changed to need manual approval of uploads, all nice.

Right now (that is, 21:38 UTC) I am mosty waiting for the rest of the people working on the release (CD builders, press, website, ) to finish their task (building hundreds of images just takes ages). When that is done we can then do a coordinated push to the world. Last plans for that would mean in about an hour . Lets see, Mr. Murphy sure can be around. Thanks go out to everyone involved in preparing this release, be it in the past by fixing bugs, uploading packages, doing whatever was needed, as well as doing the work today. Update: If you miss us setting the new testing AKA wheezy to something inside there - there is no need to. It is simply the continutation of testing with just another name, which did not get changed contents, so nothing done for that. :)

For some reason, I don t really know why, I decided to switch my keyboard layout over to something weird. No, not the french brainfuck :) that is actually possible to type on in a halfway acceptable speed (shutup Bubulle), but the Neo thing for german keyboards. Hell. I didn t know how slow one can be typing. About a letter in 3 seconds, or something like it. I think, we should add something to our mandantory set of rules for anyone on any Debian Mailinglist: Whenever a flamewar starts, everyone who participates has to switch their keyboard layout to a random new layout. For every paragraph of every mail they write. Would stop the thread before it really happens. Ohwell, lets see where I stand in a week/month from now. Maybe I can actually use it then. And maybe I really type with 10 fingers then, and not something like the 3 til 8 i currenly use (depends on moon phase how many, i think). I m not slow with my current system, far from it, but I do sometimes have pain in my hands. Maybe that goes away too if I adopt something better, with less weird hand/finger movings. Though the way over will be painful in itself.

In response to another post I think the comic says it all best. :) (Not using garfield.com directly, as that useless thing of a page isn t usable. Flash ridden fuckup.

Debian s testing distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the Constantly Usable Testing (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen. About Debian unstable & testing Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed. Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it s free of new release-critical bugs, if it s available on all supported architectures, and if it doesn t break any other package already present in testing. The Release Team (RT) controls this tool and provide hints to help it find a set of packages that can flow from unstable to testing. Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn t boot, or X that doesn t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that? Known problems with testing Disappearing software The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests. Long delays for security and important fixes Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen. This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm. Not always installable With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:

Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. [...] The problem with d-i development at the moment is more that we re very slow at producing new d-i *releases*. [...] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).

CUT s history CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian s sole product and that testing could become with some work a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years. But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic. There s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they re supposed to address the problems identified. The ideas behind CUT Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named cut ). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be rolling . Regular snapshots of testing There s general agreement that regular snapshots of testing are required: it s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest cut . For clarity, the official codename would be date based: e.g. cut-2010-09 would be the cut taken during September 2010. While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced. One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it s not necessarily so problematic as one might imagine. Testing s security record is generally better than stable s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable. Since it s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:

The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.

So if it s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release. CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users. Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users. In summary, regular snapshots address the not always installable problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem. A new rolling distribution? Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:

How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?

In Lucas s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a constant flux of new upstream releases . For him, that would be something quite unique in the Free Software world . The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution. If testing were used as the rolling distribution, the problem of disappearing packages would not be fixed. That s why there have been discussions of introducing a new distribution named rolling that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing. The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium). Then it s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing). Given the frequent releases, it s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice if made opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems). While being ahead of testing can be positive for the users, it s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed. The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name rolling does not suggest that the software is not yet ready for prime time. Conclusion Whether CUT will be implemented remains to be seen, but it s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users. The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut. From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.

This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.

21 comments Support my work

Wellwell, the meeting is over since Sunday evening, but for some reasons I only was able to send the minutes today. Go get them over at the project list while they are still fresh, before another one snatches them away from you. They are basically a short summary of what we did or intend to do soon. I do like such kind of meetings, they usually help a lot, both in getting things done but also in lifting up my spirit and motivation to do something. And so I am actually looking forward to the springtime plan we have and hope it will work out nicely, and that we get all of the team together. The more, the better. :)

And there, another day come and passed, still no real breakage, only some small things here and there. We are doing something wrong, definitely. Ok, granted, providing everyone who has experimental in their sources.list with a free upgrade path due to us suddenly forgetting that apt actually really wants this NotAutomatic: yes flag in the release instead of just looking it up in our database, later on doing the same to all of the backports user (accidently, happens when you have identical codebases), does count enough for a small icecream. But we didn t have anything serious, which is great. Alex finished patching dak rm in a way that we can now actually close all bugs attached to a (source) package we remove and was thinking of enhancing it to actually also be able to close wnpp bugs attached to it (think of O: bugs for example). Mark had design work on g-p-s, which actually is how do we generate packages and sources and how to move the config sanely into a database . Sounds easy, but trust me, it isn t, when you consider all the special cases Debian has. He also jumped in helpfully whenever there was a question around some coding and stuff, at one time turning up with the insanity/genius/younameit code that allows us to transfer even more of dak.conf into the database. This gets us a good way along the way that should end up with a dak.conf consisting of only the database details and nothing else. My job today looked similar like yesterday. Merge code I got, eat, remove lots of values from dak.conf. This weekend besides totally missing out on Baklava, had us change our dak code quite a bit: 46 files changed, 982 insertions(+), 1060 deletions(-). Besides that we do have about 18 points on our meeting notes, a mail going off to one debian lists sometime the next days summarizing it a bit more.

Hello fellow developers, The summer is over but I m happy to announce that this year s Summer of Code at Debian has been better than ever! This is indeed the 4th time we had the privilege of participating in the Google Summer of Code and each year has been a little different. This year, 8 of our 10 students succeeded in our (very strict!) final evaluations, but we have reasons to believe that they will translate into more long-term developers than ever, all thank to you. The highlight this year has been getting almost all of our students at DebConf10. Thanks again this year to generous Travel Grants from the Google Open Source Team, we managed to fly in 7 of our students (up from 3!). You certainly saw them, presenting during DebianDay, hacking on the grass of Columbia, hacking^Wcheering our Debian Project Leader throwing the inaugural pitch of a professional baseball game or hacking^Wsun-tanning on the tr s kitsch Coney Island beach. Before I give the keyboard to our Students, I d like to tell you that it will be the pleasure and honor of Obey Arthur Liu (yours truly, as Administrator) and Bastian Venthur (as Mentor) to represent Debian at the Summer of Code 2010 Mentors Summit on 23-24 October 2010, at the Google Headquarters in Mountain View. Like last year, we expect many other DDs to be present under other hats. We will be having 2 days of unconference on GSoC and free software related topics. We all look forward to reporting from California on Planet and soc-coordination@l.a.d.o! All of our students had a wonderful experience, even if they couldn t come to DebConf, that is best shared in their own voice, so without further ado, our successful projects: Multi-Arch support in APT by David Kalnischkies, mentored by Michael Vogt apt-get install MultiArch does mostly work now as most code is already merged in squeeze, but if not complain about us at deity@l.d.o! Still, a lot left on the todo list not only in APT so let us all add MultiArch again to the Release Goals and work hard on squeezing it into wheezy. Debbugs Bug Reporting and Manipulation API by David Wendt Jr., mentored by Bastian Venthur Hello, I m David Wendt, and I went to Debconf10 to learn more about the development side of Debian. Having used it since the 9th grade, I ve been intimately familiar with many of Debian s internals. However, I wanted to see the developers and other Debian users. At DebConf, I was able to see a variety of talks from Debian and Ubuntu developers. I also got to meet with my mentor as well as the maintainer of Debbugs. Content-aware Config Files Upgrading by Krzysztof Tyszecki, mentored by Dominique Dumont Config::Model is now capable of manipulating files using shorter and easier to write models. Thanks to that, packagers may start experiment with creating upgrade models. Further work is needed to support more complicated config files Dominique Dumont is working on DEP-5 parser, I ll shortly start working on a cupsd config file parser.
The best thing about DebConf10 is that every person I talked with knew what I was doing. I had a mission to get some feedback on my project. Everybody liked the idea of making upgrades less cumbersome. On the other side, it was my first visit to United States, so I decided to go on a daytrip on my own (instead of staying inside the building, despite heat warnings). I had a chance to visit many interesting places like Ground Zero, the UN headquarters, Grand Central Terminal, Times square and Rockefeller Center that was a great experience. Hurd port and de-Linux-ization of Debian-Installer by J r mie Koenig, mentored by Samuel Thibault Debconf10 was great! Among other people working on the installer, I met Aur lien Jarno from the Debian/kFreeBSD team and we worked together on a cross-platform busybox package. Besides, the talks were very interesting and I ve filled my TODO-list for the year.
For instance I learned about the Jigsaw project of OpenJDK, and how Debian would be the ideal platform to experiment with it. More generally, some people think Debian could push Java 7 forward and I d like to see this happen. Smart Upload Server for FTP Master by Petr Jasek, mentored by Joerg Jaspert I must say that it was great time for me in NY, I ve met and talked and coded with people from ftp-master team like Torsten Werner who helped me to push the project a bit further and with some other people who were looking forward to release of the tool which I hope they will use quite soon. Everybody interested, everybody excited, really cool place and time. And I can t forget the Coney Island beach and stuff, lot of fun, lot of sun;) Aptitude Qt by Piotr Galiszewski, mentored by Sune Vuorela Currently, development branches support full features searching, viewing extended package s informations, performing cache and packages operations. Code and GUI still require a lot of work which will be continued. Informations about further progress could be found on aptitude mailing list and repository rss channel. Debian-Installer on Neo FreeRunner and Handheld Devices by Thibaut Girka, mentored by Gaudenz Steinlin For me, DebConf 10 started at the airport, where Sylvestre Ledru (whom I didn t know of before) was wearing a GSoC 2007 t-shirt, that is, given the circumstances, almost equivalent to say I m a hacker, I m going to DebConf 10 .
I ve spent my time at the conference attending various talks, hacking, meeting DDs and other hackers (amongst others, my co-mentor Per Andersson, Paul Wise, Julien Cristau, Christian Perrier, Cyril Brulebois, Martin Michlmayr, Colin Watson and Otavio Salvadores who I have to thank for his patience while dealing with my questions), chatting, cross-signing keys, rushing to finish eating before 7pm, getting sunburnt, sightseeing (thanks, Arthur, for the lightning-fast tour of Manhattan!), and so on. Debian Developers and community, we count on you. See you next year! (cross-posted to debian-devel-announce@l.d.o and soc-coordination@l.a.d.o)

Wanted to post this yesterday, but somehow didnt work out, so here we go now Another meeting day passed, another lot of commits into our repository, another day without entirely breaking the system. But wait, one more chance to break it today, maybe we finally manage. :) Yesterday we got volatile sorted out for Squeeze (Lenny stays like it is, seperate). Its now a suite on ftp-master, and we have all the neccessary information and workflow around it defined. At least so far that we can operate the suite, we don t define what gets into it, we just take that from the volatile team (or, I think, in future, from the SRMs) in a kind of textfile similar to how testing works. Besides having Alex unbreak our bts categorize script, Mark fixing up our BYHAND processing and implementing the volatile stuff I mentioned above, we also merged a patchset from Luca that allows us to provide the packages.debian.org service with a tree containing changelogs, copyright files, NEWS.Debian and README.Debian. Metadata basically. Which they extracted in the past, and that was taking up quite a lot of resources. Much easier if we provide it and they just use it. This is now done for both, the main archive as well as the backports archive. As soon as I upgrade the security archive it will also be done there, which should additionally enable packages.debian.org to display those files for all archives (run by FTPMaster). This update is scheduled for the upcoming week, DSA already gave me the needed bits and pieces on a host so I can do it without breaking security during the process. We currently export this metadata set only to the packages.debian.org host. We do want to provide it to more, but I am waiting for a reply from DSA as to which machine I should use for the open access to those files. When I have that reply it should be available for you all. As a sidenote from that: If you need to compute any data that has to do with the Debian archive (information out of source/binary packages or something around it) and if you do that with your own scripts - you are most likely doing it wrong. Talk to us. We might be able to help you extracting the data you need and keep it updated. Recent examples for that are the above changelogs, but also things like the package-file-mapping snapshot uses to easily determine which file belongs to which source/version, but also information like the Maintainers index, the Uploaders index and whatnot. The simple way is - ask us. We are processing a shitload of data, it might not be too complicated to let your set fall out somewhere at the side, but if you never ask you will never find out Today we start working on some internals of dak and will have some more discussion on agenda points like data.debian.org and the git source format. We already had some words on other points, but those are ones which will take a long time more to fix, like generally reworking the internals of our database to enable us having multiple archives with pretty different access levels inside one machine. (Think of security archive on ftpmaster host. You want to ensure embargoed security issues actually stay embargoed until they are released. Means a good set of neccessary changes inside the database to handle this. We have ideas, but getting them implemented will take a bit).

---

And of course we had been out for food too, helps to stay alive. Turkish restaurant. All nice and well - until they told us they have NO BAKLAVA. Args. Suckers. They ruined this year! WAAAAAAAAAAAAAAAAH. I wish them a windows OS for their live! If not worse! (Can it be worse than windows?) ;-)

I announced it some while ago on debian-project, we have a FTPMaster meeting this weekend. It started out with a little workout session View image before we started discussing some things from our agenda. In the middle of the discussion we took a little sidestep, looking for a victim to promote to ftpmaster. So we selected someone not attending, he can t run away screaming. And for that: Send your condolences over to Torsten Werner. Or maybe congratulations, your call :) Right now Mark is fixing up our byhand processing and Alex is doing some other Debian work, while I m merging patches from Luca. Thats it for now, sometime soon we are going to eat, after which we put some more discussions on our agenda. Still two more days to go :)

It has been a pretty exhausting time since my last blog post in my area of Debian land . If you followed the news you should have seen some of it. Last weekend there was the point release. For those I have the (good?) position to sit in the middle of the working queue, where the hardest part (sorting out which stuff to include, getting it all build, etc) is already done. Still, due to the amount of steps neccessary to take its usually a multi-hour thing. This one release was even slower, as we had to wait for hours until the package lists had all been built - unfortunately the thing writing that keeps berkeley db files of data around as cache - and uses the FULL path to the files as key. How stupidly annoying when the path changes during a server move. Ohwell. The other big part that got a tiny bit of press coverage was the move from www.backports.org over to backports.debian.org. This was actually waiting for me for a good longish while already (err, well, since well before DC9 this was decided to happen) but now I finally got myself moved. This turned out to be much more work than I ever feared (which is part of the reason why it took so long), but some numbers:

• At the start the code changes between the revision running on oldbpo compared to HEAD: 303 files changed, 21227 insertions(+), 468357 deletions(-) (There had been 1723 commits in the master branch since the last time bpo got it, thats about a quarter of all we have in master!)
• Database schema version oldbpo: 5, HEAD: 35
• sub-sub-sub-optimal damn old shell scripts for the cronjobs.

The db update alone took me a week to figure out which steps to do in which order. Good to have an automated system, if you the manage to have the updates run specific to the main archive. Ups. When I was done with the database (during which I managed to just erase all work done by formorer and rhonda on the webpage, ups^2) I had some code and config cleanups to do, and especially manage to get it all running with a new path layout. Much less complicated, but enough work to do, before I finally was far enough to get it all move so far that formorer could start testing it and we could turn off the old system. At this time I had to fix up our shell scripts for our cron jobs, but that (with one exception) was a nearly straight forward easy task. And on Wednesday I managed another goal - backports is fully running the master branch of dak, the exact same code as on ftp-master. The cronjobs are slightly different, as well as the config (of course, duh), but otherwise its the same. My current goal and the last things left on this move todo is to change the cronjobs so that debian as well as backports do run the SAME scripts, but their configs list which tools should be run when. Considering that my dinstall cron already contains something pretty much a state machine I think I m insane enough to get this in shell, but when I told that to Mark (you know, the second ftpmaster :) ), I heard something very much like vomiting (you know, 700km between us, this might mean something) and then the plea to use a real language for the task. Oh my. I like my shell. :)

---

There is one big black cloud on the horizon to see. I have to do something similar for security. They are at a dak (and database) version pretty similar to what bpo had. They do need to support this obnoxious spare new format too before they can support security for squeeze, so there is no way I can ignore that archive for much longer. I already asked DSA for a vm in which I can play (not good to interrupt security archive for long), but probably wont start on this before the upcoming FTPMaster meeting. Pretty busy otherwise. Wellwell, I hope it will be less painful, as I now know the steps already. :)

I just merged a nice work from another ftpteam member, Luca Falavigna. He wrote dak make-changelog, which is intended for generating the changelogs for stable point releases. I took the chance and asked if he can extend it a bit, which he did, so today I could merge that work. Besides that we are able to generate ChangeLogs between all suites we have, we now generate a ChangeLog on every time we import a new set for testing, listing only the changes done in that run, by listing the changelog entries from the package, or in case of removed packages just listing their name with the removed version. We keep 4 rotations of that ChangeLog, you can find it (with the next mirror push) on all Debian mirrors in the dists/testing/ directory. This should be especially interesting for those people following testing on their systems, to easily see what got changed. :) Short update: We now keep the ChangeLog files with a date based name, deleting files which are older than 2880 minutes, so roughly keeping 2 days or 4 testing runs around, but in case we import more often (hey, possible short before a release), we keep more. There is a symlink ChangeLog always pointing to the latest file.

* So, here is late report! Sometime back Praveen started discussion at DebianIndia mailing list about having MiniDebConf kind of conference here in India. MiniDebConfs are generally focused on development rather than at introduction to Debian . But, since this was our first attempt, out focus was more on to introduce Debian to student community and after writing this after 5 days I realize that we ve done good job! Many thanks to Joerg Jaspert for setting up Mini website: http://in2010.mini.debconf.org/ Praveen got bunch of sponsors, many thanks to CDAC and other sponsors for making event happen with testy food and tea that kept me awake. Now what really happened at Pune? My flight was nice. I went to guest house of COEP (the host college, 2nd old engineering college in India) and took some rest and called Praveen. He was already at computer lab and setting up Virtualbox and other things. We had two places hall and lab. Unfortunately, both were at different buildings and subway between them I discussed with couple of students and we went to dinner at late night. Pro. Abhijit who was kindly give permission to use lab and college hall was another source of energy. He is handling many FOSS events in college and probably only faculty who encourage students to use and learn FOSS. The Day 1 started with long lines of registration process. As we had around 200 registration, turn out of students were 100% on first day. Hall was almost full. Dr. Shashikumar from CDAC gave nice lecture about contribution which was well received. Praveen and Onkar then took the stage and gave nice introduction about Debian. I met Onkar first time. We had really nice discussion ranging from becoming Ubuntu MOTU to his Powerbook s adapter. After that Vikram took lively session on i18n and l10n in Debian and I took boring session on Contributing to Debian . Yet another boring session from me was Developer Toolbox . Packaging workshop by me at hall made many faces to yawn heavily while Onkar and Praveen handled it nicely at Lab. Main reason for yawning was Wifi and my Macbook s capacity to deal with ndiswrapper and crappy broadcom s drivers. (Note: I should get nice machine.) I met Sham first time. I met Kushal once again and felt energetic again! Day 2 As we expected number was very low compared to Day 1. So, we closed hall and moved entire stuffs to Lab only. Vikram took again nice session on Debian Edu . It was very interactive and I should learn something from his act of presentation. Another nice thing from Day 2 was discussion with BOSS Linux team. Hold on. Lets keep some old woes aside. They are basically doing what their bosses are telling. Developers were clueless about Sending patches to upstream. Their bugzilla had Zorro bug reported. I made promise to follow-up on this and set-up patches from boss at somewhere depending upon their response. That mail is my draft let me finish it by weekend. Pavi and others took video from session. It will be nice if someone will post it somewhere We then continued on discussing bugs and packaging area. It was amazing to see Praveen become DM during conference and also to see: upstream, maintainer and sponsor in one room. I really had nice food at dinners. Mango Mastani was nice. Beer was nice. How can I forget discussion with Ramki, Ninad, Pro.Abhijit, Praveen, Pavi and others after dinner? So lets work for MiniDebConf 2011 now!

I am currently sitting somewhere around Gate A2 at Singapore Airport, waiting for my connection flight. Not my original planned one. Not the one I got first rebooked to I was in Taiwan the last 3 weeks, having had about the best time ever. The plan for the flight back was pretty simple. Today, that is Tuesday 10th August, I take the Singapore Airlines flight from Taipei to Singapore, leaving Taipei at 1810h, arrival Singapore 22:35h. I wait a little time and then board the flight from Singapore to Frankfurt, leaving at 23:50h and touchdown in Frankfurt at 06:30h. So far so easy. So far not working at all. The flight from Taipei had technical problems (they later said problems with the computer of the engines ) and was delayed by a bit more than an hour. Not enough time to get the connection flight. So the ground staff in Taipei arranged a flight for me leaving Singapore at 13:50h the next day, Hotel and Taxi and such things to be organized by Singapore ground staff. Fine fine, so over to Singapore. Got here at 23:35h, left airplane at 23:45h. Definitely not working for the original connection, so the rebooking sounded good, When I left the airplane there was ground staff waiting for me, with the information that they got another reschedule done for me. I am now supposed to fly via Zurich, getting another connection from there to Frankfurt. So over to the transfer desk to get them to handle my luggage and actually get me a usable seat. Fine fine, all worked. Leaving the transfer desk, heading over to the gate (boarding meant to start at 00:10h, so not much time left), I have the pleasure to listen to an announcement: The flight to Zurich is delayed by approximately 2 hours. Oh fun, the planned connection flight in Zurich leaves about 2 hours after the planned arrival of this flight. Back to the transfer desk, asking whats up. Simple solution, I m supposed to fly to Zurich and contact the transfer desk there, it will deal with it. Lets see what happens and if I ever get back home. :) Nice side detail - I am now supposedly getting onto an A380. Lets see if this thing is really as great as they always say. Update: Now I am home. When we started in Singapore the Flight captain announced we have good weather helping us and will end up with being only 40 minutes late in Z rich. Wellwell, I think he just used up much more gas than usual, flying faster, but whatever, it worked out, 40 minutes late it was. (I cant believe always seeing 70mph of head wind being good ). So then I had to wait at that airport. And I must say: It sucks. Taipei airport - free wifi, just some funny redirections in browser before it works. Singapore - contact a staff member of the airport, get a code, works for 6 hours, fine. Z rich - the suckers want me to pay. A lot. Useless crapshit. Way to expensive. Bad service there. The next flight then finally got me to my flights destination, Frankfurt. Had to walk through half the airport to get to the luggage, but there it actually came out first. So fine. Train home after a short visit at my work was easy, and I think I will fall asleep soonish. Ohmy. Wish I would have stayed in Taipei

Thanks to work from Luca Falavigna we can now offer the Debian removals log in a slightly easier parseable format, RFC822. That means people can now follow removals from Debian in 3 formats:

• (halfway) life using RSS showing the last few removals done
• plain old easy TEXT, where the removals.txt directly shows the current year, previous years living in their own files:
  • 2009
  • 2008
  • 2007
  • 2006
  • 2005
  • 2004
  • 2003
  • 2002
  • 2001
    as well as having one FULL
• slightly easier parseable 822 format, with the same multitude as the text files, ie.
  • 2009
  • 2008
  • 2007
  • 2006
  • 2005
  • 2004
  • 2003
  • 2002
  • 2001
    as well as having one FULL

Not enough?